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The invention relates to a method of generating a depth map comprising depth 
values representing distances to a viewer, for respective pixels of an image. 

The invention further relates to a depth map generating unit for generating a 
depth map comprising depth values representing distances to a viewer, for respective pixels 
5 of an image. 

The invention further relates to an image processing apparatus comprising: 
receiving means for receiving a signal corresponding to an image; and 
such a depth map generating unit for generating a depth map. 
The invention further relates to a computer program product to be loaded by a 
10 computer arrangement, comprising instructions to generate a depth map comprising depth 

values representing distances to a viewer, for respective pixels of an image, the computer 

arrangement comprising processing means and a memory. 

15 In order to generate a 3D impression on a multi-view display device, images 

from different virtual view points have to be rendered. This requires either multiple input 
views or some 3D or depth information to be present. This depth information can be either 
recorded, generated from multiview camera systems or generated from conventional 2D 
video material. For generating depth information from 2D video several types of depth cues 

20 can be applied: such as structure from motion, focus information, geometric shapes and 
dynamic occlusion. The aim is to generate a dense depth map, i.e. per pixel a depth value. 
This depth map is subsequently used in rendering a multi-view image to give the viewer a 
depth impression. In the article "Synthesis of multi viewpoint images at non-intermediate 
positions" by P. A. Redert, E.A. Hendriks, and J. Biemond, in Proceedings of International 

25 Conference on Acoustics, Speech, and Signal Processing, Vol. IV, ISBN 0-8186-7919-0, 
pages 2749-2752, IEEE Computer Society, Los Alamitos, California, 1997 a method of 
extracting depth information and of rendering a multi-view image on basis of the input image 
and the depth map are disclosed. 
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It is an object of the invention to provide a method of the kind described in the 
opening paragraph, which is based on a new depth cue. 

This object of the invention is achieved in that the method comprises: 
determining a contour on basis of pixel values of the image, the contour 
comprising a collection of adjacent points; 

computing curvature vectors at a number of the points; and 
assigning a first one of the depth values corresponding to the first one of the 
pixels on basis of the curvature vectors. 

The rationale behind the invention is that most objects have a sphere topology, 
i.e. they contain no holes, at least on a scale relevant within images. This holds for the 
ground, humans, telephones, cars, etc. Then, based on the curvature of the outlines of objects, 
the relative depth ordering can be determined. Since, visible objects will be closer to the 
viewer than their background. That means that pixels of an image region within a contour, 
i.e. corresponding to the outlines of an object, have to be assigned depth values which are 
lower than the pixels which are outside the contour. A relatively low depth value means that 

j1 - _ .-_ — — J! _ 1_ * _ 1- * — 1 lS 1 ^1 -i-^ ^ «M n vvrn« T\s*4t^ ■«♦»■«■»'» r» t-o-tV* Z c% iMn-I^rt J\ 

UlC UUIICSpUJLMJJLLlg UUJG^L IS IClttUVCJLJ \^JAJQCi UJ ULIW V1VW W . J^v^iiil i iiing WUttt iO jluloavw ujjmj. 

outside a contour is based on the curvature vectors. The curvature vectors are oriented 
towards decreasing depth. 

There are several known techniques in the field of image processing for 
determining contours on basis of pixel values. With pixel values is meant color and/or 
luminance values. Contours can e.g. be determined by means of edge detection, homogeneity 
calculation or based on temporal filtering. Contours can be open or closed. 

Computing curvature vectors for a curve is a well-known technique in 
mathematics. In this case the curve is a contour. A characteristic of a curvature vector of a 
particular point at a curve is that its orientation is orthogonal to the local curve segment at 
that particular point. Another characteristic of a curvature vector of a particular point at a 
curve is that its length is related to the angle between a first line segment between the 
particular point md a neighboring paint and a second line segment between JtheLperti cn l a r 
pamtHEEd Its othor nsighborfngpoinL 

F'rsfenibfcz. £S£fexdn:rlii£ firorujieijf ifiaajd^fcx^iii^a eomuTimz-.QQmnn1m.™ 
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Acquisition". This patent application has filing number EP 03 100092.0 (Attorney Docket 
number PHNL030006). For computing a first one of the depth derivatives the following 
applies: 

a size of a first one of the depth derivatives is computed on basis of the length 
of a first one of the curvature vectors; and 

a direction of a first one of the depth derivatives is computed on basis of the 
orientation of a first one of the curvature vectors. 

Computing the other depth derivatives is based on similar rules. 

An embodiment of the method according to the invention further comprises: 

computing a collection of average vectors on basis of the curvature vectors, 
the average vectors having mutually equal lengths; and 

assigning a first one of the depth values corresponding to the first one of the 
pixels on basis of the average vectors. 

As shown in Fig. 1, a circle provides a constant curvature and thus a constant 
depth derivative along its boundary with respect to its surroundings. A square however, see 
Fig. 3 provides a very non-uniform depth derivative, being maximal in the comers, and zero 
everywhere else. To get mutually equal depth values for pixels which belong to the same 
object, some smoothing of curvature vectors is needed. That means that the curvature vectors 
of a contour are transformed into a new set of average vectors. Subsequently, these average 
vectors are applied to compute depth derivatives and subsequently depth values. 

Preferably, the collection of average vectors is computed by means of parallel 

transport. 

Preferably, assigning the first one of the depth values comprises computing 
depth derivatives on basis of the respective average vector and computing the first one of the 
depth values on basis of a first one of the depth derivatives. Then, for computing a first one 
of the depth derivatives the following applies: 

a size of a first one of the depth derivatives is computed on basis of the length 
of a first one of the average vectors; and 

a direction of a first one of the depth derivatives is computed on basis of the 
orientation of a first one of the average vectors. 

Computing other depth derivatives is based on similar rules. 

The depth value which is based on curvature vectors, optionally on basis of 
average vectors, can be directly used as depth value for rendering a multi-view image, e.g. as 
described in the cited article. Preferably, the depth value according to the invention is 
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combined with other depth values which are based on alternative depth cues as mentioned 
above. 

It is a farther object of the invention to provide a depth map generating unit of 
the kind described in the opening paragraph, which is based on a new depth cue. 
5 This object of the invention is achieved in that the generating unit comprises: 

determining means for determining a contour on basis of pixel values of the 
image, the contour comprising a collection of adjacent points; 

computing means for computing curvature vectors at a number of the points; 

and 

10 - assigning means for assigning a first one of the depth values corresponding to 

the first one of the pixels on basis of the curvature vectors. 

It is a farther object of the invention to provide an image processing apparatus 
comprising a depth map generating unit of the kind described in the opening paragraph which 
is arranged to generate a depth map based on a new depth cue. 
15 This object of the invention is achieved in that the generating unit comprises: 

determining means for the dete rminin g a contour on basis of pixel values of 
the image, the contour comprising a collection of adjacent points; 

computing means for computing curvature vectors at a number of the points; 

and 

20 - assigning means for assigning a first one of the depth values corresponding to 

the first one of the pixels on basis of the curvature vectors. 

It is a fiirther object of the invention to provide a computer program product of 
the land described in the opening paragraph, which is based on a new depth cue. 

This object of the invention is achieved in that the computer program product, 
25 after being loaded, provides said processing means with the capability to cany out: 

determining a contour on basis of pixel values of the image , the contour 
comprising a collection, of adjacent points; 

computing curvature vectors, at a number of the points; and 
a ssigning a first one of the depth values -corresponding to th e fircone of the 
30 pixels on .hasisjof 1iisicur7Htorc:ve3tors. 
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These and other aspects of the depth map generating unit, of the image 
processing apparatus, of the method and of the computer program product, according to the 
invention will become apparent from and will be elucidated with respect to the 
5 implementations and embodiments described hereinafter and with reference to the 
accompanying drawings, wherein: 

Fig. 1 A schematically shows an image and the corresponding depth map being 
generated with the method according to the invention; 

Fig. IB schematically shows a portion of a contour and the curvature vector; 
10 Fig. 1C schematically shows the computation of depth values on basis of depth 

derivatives; 

Fig. 2 schematically shows the computation of average vectors on basis of 
curvature vectors; 

Fig. 3 schematically shows another image and the corresponding depth map 
1 5 being generated with the method according to the invention; 

Fig. 4 schematically shows a multi-view image generation unit comprising a 
depth map generation unit according to the invention; and 

Fig. 5 schematically shows an embodiment of the image processing apparatus 
according to the invention. 
20 Same reference numerals are used to denote similar parts throughout the 

figures. 

Fig. 1 A schematically shows an image 100 and the corresponding depth map 
25 122 being generated with the method according to the invention. Fig. 1 shows an image 100 
representing a first object 104 and a second object 102 which is located behind the first object 
104. It should be noted that the background can form one or more objects, e.g. the sky or a 
forest or a meadow. 

Fig. 1A shows a depth map 122 comprising a first group 1 16 of depth values 
30 corresponding to the first object 104 and comprises a second group 1 18 of depth values 
corresponding to the second object 102. The depth values of the first group 1 16 of depth 
values are lower than the depth values of the second group 1 18 of depth values, meaning that 
the first object 104 is more close to a viewer of the image 100 or to a multi-view image 
which is based on the image 100, than the second object 102. 
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The depth map 122 is generated on basis of the method according to the 
invention. For the generation of depth values the following steps are performed: 
A: a contour 106 is determined on basis of pixel values of the image 100. In this 

case, the contour 106 corresponds to the edge of the first object 104. The contour is found by 
5 means of edge detection. The contour 106 comprises a collection of adjacent points. These 
adjacent points can correspond with the pixel coordinates. Alternatively these points are 
expressed in floating points related to the pixel matrix of the image 100; 
B: curvature vectors 108-1 14 are computed for a number of the points of the 

collection. Typically, a particular curvature vector 112 is computed for a particular point 126 
10 on basis of the coordinates of the particular point 126 and the respective coordinates of the 
neighboring points 124 and 128. See the description in connection with Fig. IB. 
C: depth derivatives are computed on basis of the curvature vectors 108-1 14. The 

value of the depth derivatives is related to the length of the respective curvature vectors and 
the direction of the depth derivatives is based on the orientation of the respective curvature 

— E 

15 vectors. In Equation 1 is specified how a depth derivative VD(p) for a point p is computed 
on basis of the corresponding curvature vector C(p) for the point p . The value a is a 
constant. 

VD(p) = -aC(p) (1) 
See the description in connection with Fig. 1C for the computation of depth 
20 values on basis of depth derivatives. 

Fig. IB schematically shows a portion of a contour 106 and the particular 
curvature vector 112 which belongs to this portion of the contour 106. The contour 106 
comprises a collection of adjacent points of which three points 124, 126. 128 are depicted in 
Fig. IB. The orientation of the particular curvature 1 12 is orthogonal to the portion of the 
25 contour at the particular point 126. Another characteristic of the particular curvature vector 
112 of the particulaiLpoint 126 is that its length is related to the angle between a first line 
segment 130 between the particular_pointl26 and a neighboring point 124 and a second line 
segment 132 between the paraeulaiLpomt 126 and its other neighboring point 128. 

Fig. 1C schematically s'frov/s the computation of depth values on "basis of depth 
30 derfezsfe^ a fi^dopthj^ 
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with filing number EP 03 100092.0. The value of the particular depth derivative VD(p) = -3, 
i.e. length 3 and having a direction opposite to the curvature vector. For an origin pixel 140 
the depth value has been set to a well chosen predetermined offset depth value which in this 
case is equal to 10. The computation of the first depth value corresponding to the first pixel 
5 142 is relatively easy then, i.e. 10-3=7. 

Fig. 2 schematically shows the computation of average vectors 222-236 on 
basis of curvature vectors 202-216. This computation is based on so-called parallel transport 
of vectors along a curve. The computation comprises the following steps: 
D: contour 106 is unwind into a straight line 208. While unwinding, the length of 

10 the curvature vectors 202-21 6 and their direction with respect to the contour 106 are 

maintained. By definition, the unwound curvature vectors point either up or down. That 
means that each of them can be represented by means of a real number. 
E: then the unwound curvature vectors, i.e. real numbers are accumulated and 

optionally divided by the number of curvature vectors or by a predetermined constant value 
15 (e.g. 2% ) , resulting into average vectors 222-236. 

F: then the original curvature vectors 202-21 6 are replaced by the average vectors 

222-236.The average vectors 202-216 have the following characteristics: 
they all have the same length; 

each of them is oriented orthogonal to its local curve segment; 

20 - each of them is oriented parallel to its corresponding original curvature vector. 

That means that it either directs in the same direction or in the opposite direction. 

Fig. 3 schematically shows another image 300 and the corresponding depth 
map 324 being generated with the method according to the invention. Fig. 3 shows an image 
300 representing a first object 104, i.e. a round object, a third object object 105, i.e. a square 

25 object, which is located behind the first object 104 and a second object 102, i.e. the 
background which is located behind the third object 105. 

Fig. 3 shows a depth map 324 comprising a first group 328 of depth values 4 
corresponding to the first object 104 and comprises a second group 327 of depth values 
corresponding to the second object 102. The depth map 324 farther comprises a third group 

30 326 of depth values corresponding to the third object 105. The depth values of the first group 
328 of depth values are lower than the depth values of the second group 327 of depth values, 
meaning that the first object 104 is more close to a viewer of the image 300 or to a multi- 
view image which is based on the image 300, than the second object 102. 
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The depth map 324 is generated on basis of the method according to the 
invention. For the generation of depth values the following steps are performed: 
A: the three open contours 302-306 are determined on basis of pixel values of the 

image 300. In this case, the first contour 302 corresponds to a portion of the edge of the first 
5 object 104, the second contour 304 corresponds to a portion of the edges of the third object 
105 and the third contour 306 corresponds to another portion of the edge of the first object. 
These three open contours 302-306 are determined on basis of edge detection followed by 
detection of T-junctions. 

B: then sets of curvature vectors 308-312 are computed for the respective open 

10 contours 302-306. As can be seen, the curvature vectors 3 10 corresponding to the second 
contour 304 are relatively high. 

E: then sets of average vectors 3 1 8-322 are computed for the respective sets of 

curvature vectors 308-312 as described in connection with Fig. 2; 

C: then depth derivatives are computed on basis of the average vectors 3 1 8-322. 

15 The value of the depth derivatives is related to the length of the respective average vectors 

and the direction of the depth derivatives is based on the orientation of the respective average 
vectors. The depth values are computed on basis of the depth derivatives as disclosed in the 
patent application with filing number EP 03100092.0. In Equation 2 is specified how a depth 
derivative VZ)(» for a point p is computed on basis of the corresponding average vector 

20 C average (p) for the point p . The value a is a constant. 

VD(p) ^ -a C average (p) (2) 

Fig 4 schematically shows a multi-view image generation unit 400 comprising 
a depth map generation unit 401 according to the invention. The multi-view image generation 
unit 400 is arranged to generate a sequence of multi-view images on basis of a sequence of 

25 video images. The multi-view image generation unit 400 is provided with a stream of video 
imag es at the input connector 408 and provides two correlated streamsof video imagesat the 
output connectors 410 and 412, respectively. These two-correlated streams of video images 
are_to be providedto a mulli=view~ display dexdoe. which is ammgsd to visualise, a. first series, 
of ^fews" oii- basis of 'the first one of the corrdatsd-Stei^s of vidso images andto visualizes 

30 Becond-sssissrof vmwB'on. b.asis_of thoisesondjon^of ctoimprof wdaarimsgesr 

Ira iirr ? La, ^rwsr. sbs^rr~~ tii~ 'i^rr sirbs::c v?3wrrg7 Ms izic j^TtsiAths r^^ords^n^-cnL 



PHNL040243EPP 

9 12.03.2004 
and that the second one of the correlated streams of video images is rendered on basis of the 
sequence of video images as received. Preferably, both streams of video images are rendered 
on basis of the sequence of video images image as received. The rendering is e.g. as 
described in the article "Synthesis of multi viewpoint images at non-intermediate positions" 
5 by P. A. Redert, E.A. Hendriks, and J. Biemond, in Proceedings of International Conference 
on Acoustics, Speech, and Signal Processing, Vol. IV, ISBN 0-8186-7919-0, pages 2749- 
2752, IEEE Computer Society, Los Alamitos, California, 1997. Alternatively, the rendering 
is as described in "High-quality images from 2.5D video", by R.P. Berretty and F.E. Ernst, in 
Proceedings Eurographics, Granada, 2003, Short Note 124. 
10 The multi-view image generation unit 400 comprises: 

a depth map generation unit 401 for generating depth maps for the respective 
input images on basis of the transitions in the image; and 

a rendering unit 406 for rendering the multi-view images on basis of the input 
images and the respective depth maps, which are provided by the depth map generation unit 
15 401. 

The depth map generating unit 401 for generating depth maps comprising 
depth values representing distances to a viewer, for respective pixels of the images, 
comprises: 

a determining unit 402 for the determining a contour on basis of pixel values 
20 of the image 100, the contour comprising a collection of adjacent points; 

computing unit 403 for computing curvature vectors at a number of the points; 

and 

an assigning unit 404 for assigning a first one of the depth values 
corresponding to the first one of the pixels on basis of the curvature vectors 108-1 14. 

25 The determining unit 402, the computing unit 403, the assigning unit 404 and 

the rendering unit 406 may be implemented using one processor. Normally, these functions 
are performed under control of a software program product. During execution, normally the 
software program product is loaded into a memory, like a RAM, and executed from there. 
The program may be loaded from a background memory, like a ROM, hard disk, or 

30 magnetically and/or optical storage, or may be loaded via a network like Internet. Optionally 
an application specific integrated circuit provides the disclosed functionality. 

It should be noted that, although the multi-view image generation unit 400 as 
described in connection with Fig. 4 is designed to deal with video images, alternative 
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embodiments of the depth map generation unit according to the invention are arranged to 
generate depth maps on basis of individual images, i.e. still pictures. 

Fig. 5 schematically shows an embodiment of the image processing apparatus 
500 according to the invention, comprising: 
5 a receiving unit 502 for receiving a video signal representing input images; 

a multi-view image generation unit 401 for generating multi-view images on 
basis of the received input images, as described in connection with Fig 4; and 

a multi-view display device 506 for displaying the multi-view images as 
provided by the multi-view image generation unit 401. 

10 The video signal may be a broadcast signal received via an antenna or cable 

but may also be a signal from a storage device like a VCR (Video Cassette Recorder) or 
Digital Versatile Disk (DVD). The signal is provided at the input connector 510. The image 
processing apparatus 500 might e.g. be a TV. Alternatively the image processing apparatus 
500 does not comprise the optional display device but provides the output images to an 

15 apparatus that does comprise a display device 506. Then the image processing apparatus 500 
might be e.g. a set top box, a satellite-tuner, a VCR player, a DVD player or recorder. 
Optionally the image processing apparatus 500 comprises storage means, like a hard-disk or 
means for storage on removable media, e.g. optical disks. The image processing apparatus 
500 might also be a system being applied by a film-studio or broadcaster. 

20 It should be noted that the above-mentioned embodiments illustrate rather than 

limit the invention and that those skilled in the art will be able to design alternative 
embodiments without departing from the scope of the appended claims. In the claims, any 
reference signs placed between parentheses shall not be constructed as limiting the claim. 
The word 'comprising' does not exclude the presence of elements or steps not listed in a 

25 cla im. The word "a" or "an" preceding an element does not exclude the presence of a 
plurality of such elements. The invention can be implemented by means of hardware 
comprising several distinct elements and by means of a suitable programmed computer, hi 
the unit claims enumerating several means, several of these means can be embodied by one 
andlhe_same item of hardware. The umge of the_ words Ors% second and diird, etcetera do not 

30 indicsi^anjrordering^ These Tvords-zxe to be interpreted as names. 
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CLAIMS: 



1 . A method of generating a depth map (122) comprising depth values 

representing distances to a viewer, for respective pixels of an image (100), the method 
comprising: 

determining a contour (106) on basis of pixel values of the image (100), the 
5 contour (106) comprising a collection of adjacent points; 

computing curvature vectors (108-1 14) at a number of the points; and 
assigning a first one of the depth values corresponding to the first one of the 
pixels on basis of the curvature vectors (108-114). 

10 2. A method of generating a depth map (122) as claimed in Claim 1 , whereby 

assigning the first one of the depth values comprises computing depth derivatives on basis of 
the respective curvature vectors (108-1 14) and computing the first one of the depth values on 
basis of a first one of the depth derivatives. 

15 3. A method of generating a depth map (122) as claimed in Claim 2, whereby a 

size of a first one of the depth derivatives is computed on basis of the length of a first one of 
the curvature vectors. 

4. A method of generating a depth map (122) as claimed in Claim 2, whereby a 
20 direction of a first one of the depth derivatives is computed on basis of the orientation of a 

first one of the curvature vectors. 

5. A method of generating a depth map (324) as claimed in Claim 1, the method 
further comprising: 

25 - computing a collection of average vectors (320) on basis of the curvature 

vectors (310), the average vectors (320) having mutually equal lengths; and 

assigning a first one of the depth values corresponding to the first one of the 
pixels on basis of the average vectors (320). 
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6. A method of generating a depth map (324) as claimed in Claim 5, whereby the 
collection of average vectors (320) is computed by means of parallel transport. 

7. A method of generating a depth map (324) as claimed in Claim 5, whereby 

5 assigning the first one of the depth values comprises computing depth derivatives on basis of 
the respective average vector (320) and computing the first one of the depth values on basis 
of a first one of the depth derivatives. 

8. A method of generating a depth map (1 22) as claimed in Claim 7, whereby a 
10 size of a first one of the depth derivatives is computed on basis of the length of a first one of 

the average vectors (320). 

9. A method of generating a depth map (122) as claimed in Claim 7, whereby a 
direction of a first one of the depth derivatives is computed on basis of the orientation of a 

1 5 first one of the average vectors (320). 

10. A depth map generating unit (401) for generating a depth map (122) 
comprising depth values representing distances to a viewer, for respective pixels of an image 
(100), the depth map generating unit (401) comprising: 

20 - determining means (402) for the determining a contour (1 06) on basis of pixel 

values of the image (100), the contour (106) comprising a collection of adjacent points; 

computing means (403) for computing curvature vectors (108-1 14) at a 
number of the points; and 

assi gnin g means (404) for assigning a first one of the depth values 
25 corresponding to the first one of the pixels on basis of the curvature vectors (108-1 14). 

1 1 . An imag e processing apparatus (500) comprising: 

receiving means. (502) for receiving a signal corresponding to a n ima ge (100); 

and 

30 ~ a depth aisp :geasrating: tmk (40 ] ) forgenmming a depth map (122V as 
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distances to a viewer, for respective pixels of an image (100), the computer arrangement 
comprising processing means and a memory, the computer program product, after being 
loaded, providing said processing means with the capability to carry out: 

determining a contour (106) on basis of pixel values of the image (100), the 
contour (106) comprising a collection of adjacent points; 

computing curvature vectors (108-1 14) at a number of the points; and 
assigning a first one of the depth values corresponding to the first one of the 
pixels on basis of the curvature vectors (108-1 14). 



PHNL040243EPP 

14 12.03.2004 

ABSTRACT: 



A method of generating a depth map (122) comprising depth values 
representing distances to a viewer, for respective pixels of an image (100), is disclosed. The 
method comprises: detemuning a contour (106) on basis of pixel values of the image (100), 
the contour comprising a collection of adjacent points; computing curvature vectors (108- 
1 14) at a number of the points; and assigning a first one of the depth values corresponding to 
the first one of the pixels on basis of the curvature vectors (108-1 14). 



Fig. 1A 
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